In conventional virtual reality applications, visual simulation is primarily used to provide the end user with the visual illusion of interacting with the virtual environment. In mixed reality scenarios where the user jointly interacts with real objects in the physical environment (real environment) and related virtual environments, the sense of touch significantly eases navigation inside the virtual environment (VE), as users cannot pass through apparent solid surfaces, and thus mixed reality improves both the level of immersion and the skill acquisition. Often the physical contact is mimicked with an active haptic feedback device, such as two haptic devices arranged with a mobile member in the arthroscopy simulator described in US patent application publication 2010/0086905. That simulator includes a human anatomy model of a joint or organ in real size and a simulated medical instrument that imitates the real medical procedure instrument; the model is further adapted with sensors and mobile members for guiding, tracking, and controlling the medical instrument operation within the anatomy model. However, active haptic feedback is expensive from both a commercial and a computational point-of-view, in particular if stiff contact is being simulated. The degree of possible motion of the tools due to the mechanical configuration of the device is limited; they also suffer from various mechanical limitations, such as friction and inertia, and the maximum available amount of forces and torques is often not sufficient. In addition, today's devices are quite large and cannot be integrated easily into the simulator hardware. When different training scenarios have to be supported, for instance in a multipurpose training room, a different hardware setup is required for each different training model. Last but not least, from an operational point of view, active haptic feedback systems require specific setup and calibration prior to being operated; and in practice, the setup is not stable over time, thus requiring regular maintenance to avoid inducing training errors.
Passive haptic solutions, also known as tactile augmentation or mixed reality, provide a cost-effective alternative to active haptic feedback. The passive haptic technique involves the addition of inert real physical objects into the virtual environment simulator, resulting in a compelling tactile sensation. Compared to active haptic feedback, passive haptic results in a very realistic tactile sensation, in particular in combination with a rigid environment. However, the modifications of the virtual environment require modifications of the physical counterparts, as the mapping between the virtual environment and the associated physical objects is usually one-to-one to avoid a mismatch between the visual and tactile cues. This one-to-one mapping to the underlying physical environment makes the virtual environment inflexible. In the context of passive haptic surgical simulators, this may hinder simulating different virtual patients within the simulator or changing the training scenario dynamically.
This limitation will be better understood with reference to FIG. 1. FIG. 1 represents two different mixed reality scenarios B and C simulated from the real-world A interaction of a user physical manipulator 100a (Hp) with a physical object 110a (Ep) in the real environment A. The user physical manipulator 100a (Hp) may be represented by a manipulator avatar 100b, 100c (Hv) while the physical object 110a (Ep) may be represented by a physical object avatar 110b, 110c (Ev) in the virtual environment according to two different scenarios B and C. In scenario B the physical object avatar 110b (Ev) is larger than the real object 110a while the manipulator avatar 100b (Hv) has the same size has the original one 100a, so the default user interaction results in a collision and penetration of the manipulator avatar 100b with the object avatar 110b at discrepant contact interaction point 120b in the virtual environment B. The collision is seen when point 120b on the tip of the manipulator avatar touches the object avatar 110b (Ev) on the screen before it is experienced with the hand. Thus, no haptic stimuli are received at this point, which disturbs the user because of the resulting asynchronous visual and haptic stimuli perception. Conversely, in scenario C the physical object avatar 110c (Ev) is smaller than the real object 110a, so the end user does not see the manipulator avatar 100c (Hv) colliding with the object avatar 110c at discrepant contact interaction point 120c in the virtual environment C, while the end user feels the collision from the real object interaction as the real manipulator 100a is blocked at contact point 120a in the real environment, which results in a non-realistic mixed reality experience.
To overcome this limitation, recent work has proposed space warping techniques wherein the virtual space is distorted such that a variety of virtual environments can be mapped onto one single physical environment by allowing some discrepancy in the mapping between the virtual and the physical objects. To this end, the virtual space is warped, resulting in a discrepancy between the motion of the user-guided instrument in the real and in the virtual world, which may not be noticed by the end user as long as it is under his/her perceptual threshold. As will be recognized by those skilled in the art, warping may be accomplished by representing a 3D model by its surfaces, for instance a polygonal surface representation, manipulating the representation, and converting back the manipulated representation into a modified 3D model, but other warping techniques are also possible. FIG. 2 illustrates such an example where a user-guided instrument 100a moves with constant velocity on a flat plane 110a in the real environment A, and the flat plane is warped into a curved surface 110b in the virtual environment B. In turn, the velocity of the corresponding user-guided instrument avatar 100b in the virtual environment is not constant in order to maintain a coherent sensation during the movement. In “Redirected touching: warping space to remap passive haptics”, published in IEEE Symposium on 3D User Interfaces, pp. 129-130, 2010, Kohli proposed to enhance virtual reality training systems with space warping in order to increase the number of training scenarios over a given real environment. However, it is not adaptive, in particular when the virtual environment parameters change over time and/or space according to the mixed reality scenario to be simulated.